{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# nuScenes devkit tutorial\n",
    "\n",
    "Welcome to the nuScenes tutorial. This demo assumes the database itself is available at `/data/sets/nuscenes`, and loads a mini version of the full dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A Gentle Introduction to nuScenes\n",
    "\n",
    "In this part of the tutorial, let us go through a top-down introduction of our database. Our dataset comprises of elemental building blocks that are the following:\n",
    "\n",
    "1. `log` - Log information from which the data was extracted.\n",
    "2. `scene` - 20 second snippet of a car's journey.\n",
    "3. `sample` - An annotated snapshot of a scene at a particular timestamp.\n",
    "4. `sample_data` - Data collected from a particular sensor.\n",
    "5. `ego_pose` - Ego vehicle poses at a particular timestamp.\n",
    "6. `sensor` - A specific sensor type.\n",
    "7. `calibrated sensor` - Definition of a particular sensor as calibrated on a particular vehicle.\n",
    "8. `instance` - Enumeration of all object instance we observed.\n",
    "9. `category` - Taxonomy of object categories (e.g. vehicle, human). \n",
    "10. `attribute` - Property of an instance that can change while the category remains the same.\n",
    "11. `visibility` - Fraction of pixels visible in all the images collected from 6 different cameras.\n",
    "12. `sample_annotation` - An annotated instance of an object within our interest.\n",
    "13. `map` - Map data that is stored as binary semantic masks from a top-down view.\n",
    "\n",
    "The database schema is visualized below. For more information see the [nuScenes schema](https://github.com/nutonomy/nuscenes-devkit/blob/master/docs/schema_nuscenes.md) page.\n",
    "![](https://www.nuscenes.org/public/images/nuscenes-schema.svg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Google Colab (optional)\n",
    "\n",
    "<br>\n",
    "<a href=\"https://colab.research.google.com/github/nutonomy/nuscenes-devkit/blob/master/python-sdk/tutorials/nuscenes_tutorial.ipynb\">\n",
    "    <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\" align=\"left\">\n",
    "</a>\n",
    "<br>\n",
    "\n",
    "If you are running this notebook in Google Colab, you can uncomment the cell below and run it; everything will be set up nicely for you. Otherwise, manually set up everything."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !mkdir -p /data/sets/nuscenes  # Make the directory to store the nuScenes dataset in.\n",
    "\n",
    "# !wget https://www.nuscenes.org/data/v1.0-mini.tgz  # Download the nuScenes mini split.\n",
    "\n",
    "# !tar -xf v1.0-mini.tgz -C /data/sets/nuscenes  # Uncompress the nuScenes mini split.\n",
    "\n",
    "# !pip install nuscenes-devkit &> /dev/null  # Install nuScenes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "from nuscenes.nuscenes import NuScenes\n",
    "\n",
    "nusc = NuScenes(version='v1.0-mini', dataroot='/data/sets/nuscenes', verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A look at the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. `scene`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "nuScenes is a large scale database that features annotated samples across ***1000 scenes*** of approximately 20 seconds each. Let's take a look at the scenes that we have in the loaded database."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nusc.list_scenes()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at a scene metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_scene = nusc.scene[0]\n",
    "my_scene"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. `sample`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In scenes, we annotate our data every half a second (2 Hz).\n",
    "\n",
    "We define `sample` as an ***annotated keyframe of a scene at a given timestamp***. A keyframe is a frame where the time-stamps of data from all the sensors should be very close to the time-stamp of the sample it points to.\n",
    "\n",
    "Now, let us look at the first annotated sample in this scene."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "first_sample_token = my_scene['first_sample_token']\n",
    "\n",
    "# The rendering command below is commented out because it tends to crash in notebooks\n",
    "# nusc.render_sample(first_sample_token)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's examine its metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "my_sample = nusc.get('sample', first_sample_token)\n",
    "my_sample"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A useful method is  `list_sample()` which lists all related `sample_data` keyframes and `sample_annotation` associated with a `sample` which we will discuss in detail in the subsequent parts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "nusc.list_sample(my_sample['token'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. `sample_data`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The nuScenes dataset contains data that is collected from a full sensor suite. Hence, for each snapshot of a scene, we provide references to a family of data that is collected from these sensors. \n",
    "\n",
    "We provide a `data` key to access these:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_sample['data']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the keys are referring to the different sensors that form our sensor suite. Let's take a look at the metadata of a `sample_data` taken from `CAM_FRONT`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sensor = 'CAM_FRONT'\n",
    "cam_front_data = nusc.get('sample_data', my_sample['data'][sensor])\n",
    "cam_front_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also render the `sample_data` at a particular sensor. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nusc.render_sample_data(cam_front_data['token'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. `sample_annotation`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`sample_annotation` refers to any ***bounding box defining the position of an object seen in a sample***. All location data is given with respect to the global coordinate system. Let's examine an example from our `sample` above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_annotation_token = my_sample['anns'][18]\n",
    "my_annotation_metadata =  nusc.get('sample_annotation', my_annotation_token)\n",
    "my_annotation_metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also render an annotation to have a closer look."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nusc.render_annotation(my_annotation_token)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. `instance`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Object instance are instances that need to be detected or tracked by an AV (e.g a particular vehicle, pedestrian). Let us examine an instance metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_instance = nusc.instance[599]\n",
    "my_instance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We generally track an instance across different frames in a particular scene. However, we do not track them across different scenes. In this example, we have 16 annotated samples for this instance across a particular scene."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "instance_token = my_instance['token']\n",
    "nusc.render_instance(instance_token)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An instance record takes note of its first and last annotation token. Let's render them"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"First annotated sample of this instance:\")\n",
    "nusc.render_annotation(my_instance['first_annotation_token'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Last annotated sample of this instance\")\n",
    "nusc.render_annotation(my_instance['last_annotation_token'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. `category`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A `category` is the object assignment of an annotation.  Let's look at the category table we have in our database. The table contains the taxonomy of different object categories and also list the subcategories (delineated by a period). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nusc.list_categories()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A category record contains the name and the description of that particular category."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nusc.category[9]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Refer to `instructions_nuscenes.md` for the definitions of the different categories."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7. `attribute`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An `attribute` is a property of an instance that may change throughout different parts of a scene while the category remains the same. Here we list the provided attributes and the number of annotations associated with a particular attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nusc.list_attributes()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at an example how an attribute may change over one scene"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_instance = nusc.instance[27]\n",
    "first_token = my_instance['first_annotation_token']\n",
    "last_token = my_instance['last_annotation_token']\n",
    "nbr_samples = my_instance['nbr_annotations']\n",
    "current_token = first_token\n",
    "\n",
    "i = 0\n",
    "found_change = False\n",
    "while current_token != last_token:\n",
    "    current_ann = nusc.get('sample_annotation', current_token)\n",
    "    current_attr = nusc.get('attribute', current_ann['attribute_tokens'][0])['name']\n",
    "    \n",
    "    if i == 0:\n",
    "        pass\n",
    "    elif current_attr != last_attr:\n",
    "        print(\"Changed from `{}` to `{}` at timestamp {} out of {} annotated timestamps\".format(last_attr, current_attr, i, nbr_samples))\n",
    "        found_change = True\n",
    "\n",
    "    next_token = current_ann['next']\n",
    "    current_token = next_token\n",
    "    last_attr = current_attr\n",
    "    i += 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8. `visibility`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`visibility` is defined as the fraction of pixels of a particular annotation that are visible over the 6 camera feeds, grouped into 4 bins."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nusc.visibility"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at an example `sample_annotation` with 80-100% visibility"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "anntoken = 'a7d0722bce164f88adf03ada491ea0ba'\n",
    "visibility_token = nusc.get('sample_annotation', anntoken)['visibility_token']\n",
    "\n",
    "print(\"Visibility: {}\".format(nusc.get('visibility', visibility_token)))\n",
    "nusc.render_annotation(anntoken)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at an example `sample_annotation` with 0-40% visibility"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "anntoken = '9f450bf6b7454551bbbc9a4c6e74ef2e'\n",
    "visibility_token = nusc.get('sample_annotation', anntoken)['visibility_token']\n",
    "\n",
    "print(\"Visibility: {}\".format(nusc.get('visibility', visibility_token)))\n",
    "nusc.render_annotation(anntoken)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9. `sensor`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The nuScenes dataset consists of data collected from our full sensor suite which consists of:\n",
    "- 1 x LIDAR, \n",
    "- 5 x RADAR, \n",
    "- 6 x cameras, "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "nusc.sensor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Every `sample_data` has a record on which `sensor` the data is collected from (note the \"channel\" key)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "nusc.sample_data[10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 10. `calibrated_sensor`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`calibrated_sensor` consists of the definition of a particular sensor (lidar/radar/camera) as calibrated on a particular vehicle. Let us look at an example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "nusc.calibrated_sensor[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the `translation` and the `rotation` parameters are given with respect to the ego vehicle body frame. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 11. `ego_pose`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`ego_pose` contains information about the location (encoded in `translation`) and the orientation (encoded in `rotation`) of the ego vehicle, with respect to the global coordinate system."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nusc.ego_pose[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the number of `ego_pose` records in our loaded database is the same as the number of `sample_data` records. These two records exhibit a one-to-one correspondence."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 12. `log`\n",
    "\n",
    "The `log` table contains log information from which the data was extracted. A `log` record corresponds to one journey of our ego vehicle along a predefined route. Let's check the number of logs and the metadata of a log."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Number of `logs` in our loaded database: {}\".format(len(nusc.log)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nusc.log[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that it contains a variety of information such as the date and location of the log collected. It also gives out information about the map from where the data was collected. Note that one log can contain multiple non-overlapping scenes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 13. `map`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Map information is stored as binary semantic masks from a top-down view. Let's check the number of maps and metadata of a map."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"There are {} maps masks in the loaded dataset\".format(len(nusc.map)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "nusc.map[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## nuScenes Basics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's get a bit technical.\n",
    "\n",
    "The NuScenes class holds several tables. Each table is a list of records, and each record is a dictionary. For example the first record of the category table is stored at:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nusc.category[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The category table is simple: it holds the fields `name` and `description`. It also has a `token` field, which is a unique record identifier. Since the record is a dictionary, the token can be accessed like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat_token = nusc.category[0]['token']\n",
    "cat_token"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you know the `token` for any record in the DB you can retrieve the record by doing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nusc.get('category', cat_token)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_As you can notice, we have recovered the same record!_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OK, that was easy. Let's try something harder. Let's look at the `sample_annotation` table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nusc.sample_annotation[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This also has a `token` field (they all do). In addition, it has several fields of the format [a-z]*\\_token, _e.g._ instance_token. These are foreign keys in database terminology, meaning they point to another table. \n",
    "Using `nusc.get()` we can grab any of these in constant time. For example, let's look at the visibility record."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nusc.get('visibility', nusc.sample_annotation[0]['visibility_token'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The visibility records indicate how much of an object was visible when it was annotated.\n",
    "\n",
    "Let's also grab the `instance_token`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "one_instance = nusc.get('instance', nusc.sample_annotation[0]['instance_token'])\n",
    "one_instance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This points to the `instance` table. This table enumerate the object _instances_ we have encountered in each \n",
    "scene. This way we can connect all annotations of a particular object.\n",
    "\n",
    "If you look carefully at the README tables, you will see that the sample_annotation table points to the instance table, \n",
    "but the instance table doesn't list all annotations that point to it. \n",
    "\n",
    "So how can we recover all sample_annotations for a particular object instance? There are two ways:\n",
    "\n",
    "1. `Use nusc.field2token()`. Let's try it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ann_tokens = nusc.field2token('sample_annotation', 'instance_token', one_instance['token'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This returns a list of all sample_annotation records with the `'instance_token'` == `one_instance['token']`. Let's store these in a set for now"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "ann_tokens_field2token = set(ann_tokens)\n",
    "\n",
    "ann_tokens_field2token"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `nusc.field2token()` method is generic and can be used in any similar situation.\n",
    "\n",
    "2. For certain situation, we provide some reverse indices in the tables themselves. This is one such example. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The instance record has a field `first_annotation_token` which points to the first annotation in time of this instance. \n",
    "Recovering this record is easy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ann_record = nusc.get('sample_annotation', one_instance['first_annotation_token'])\n",
    "ann_record"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can traverse all annotations of this instance using the \"next\" field. Let's try it. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ann_tokens_traverse = set()\n",
    "ann_tokens_traverse.add(ann_record['token'])\n",
    "while not ann_record['next'] == \"\":\n",
    "    ann_record = nusc.get('sample_annotation', ann_record['next'])\n",
    "    ann_tokens_traverse.add(ann_record['token'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's assert that we recovered the same ann_records as we did using nusc.field2token:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(ann_tokens_traverse == ann_tokens_field2token)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reverse indexing and short-cuts\n",
    "\n",
    "The nuScenes tables are normalized, meaning that each piece of information is only given once.\n",
    "For example, there is one `map` record for each `log` record. Looking at the schema you will notice that the `map` table has a `log_token` field, but that the `log` table does not have a corresponding `map_token` field. But there are plenty of situations where you have a `log`, and want to find the corresponding `map`! So what to do? You can always use the `nusc.field2token()` method, but that is slow and inconvenient. We therefore add reverse mappings for some common situations including this one.\n",
    "\n",
    "Further, there are situations where one needs to go through several tables to get a certain piece of information. \n",
    "Consider, for example, the category name (e.g. `human.pedestrian`) of a `sample_annotation`. The `sample_annotation` table doesn't hold this information since the category is an instance level constant. Instead the `sample_annotation` table points to a record in the `instance` table. This, in turn, points to a record in the `category` table, where finally the `name` fields stores the required information.\n",
    "\n",
    "Since it is quite common to want to know the category name of an annotation, we add a `category_name` field to the `sample_annotation` table during initialization of the NuScenes class.\n",
    "\n",
    "In this section, we list the short-cuts and reverse indices that are added to the `NuScenes` class during initialization. These are all created in the `NuScenes.__make_reverse_index__()` method."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reverse indices\n",
    "We add two reverse indices by default.\n",
    "* A `map_token` field is added to the `log` records.\n",
    "* The `sample` records have shortcuts to all `sample_annotations` for that record as well as `sample_data` key-frames. Confer `nusc.list_sample()` method in the previous section for more details on this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Shortcuts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The sample_annotation table has a \"category_name\" shortcut."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Using shortcut:_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "catname = nusc.sample_annotation[0]['category_name']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Not using shortcut:_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ann_rec = nusc.sample_annotation[0]\n",
    "inst_rec = nusc.get('instance', ann_rec['instance_token'])\n",
    "cat_rec = nusc.get('category', inst_rec['category_token'])\n",
    "\n",
    "print(catname == cat_rec['name'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The sample_data table has \"channel\" and \"sensor_modality\" shortcuts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Shortcut\n",
    "channel = nusc.sample_data[0]['channel']\n",
    "\n",
    "# No shortcut\n",
    "sd_rec = nusc.sample_data[0]\n",
    "cs_record = nusc.get('calibrated_sensor', sd_rec['calibrated_sensor_token'])\n",
    "sensor_record = nusc.get('sensor', cs_record['sensor_token'])\n",
    "\n",
    "print(channel == sensor_record['channel'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Visualizations\n",
    "\n",
    "We provide list and rendering methods. These are meant both as convenience methods during development, and as tutorials for building your own visualization methods. They are implemented in the NuScenesExplorer class, with shortcuts through the NuScenes class itself."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### List methods\n",
    "There are three list methods available."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. `list_categories()` lists all categories, counts and statistics of width/length/height in meters and aspect ratio."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nusc.list_categories()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. `list_attributes()` lists all attributes and counts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nusc.list_attributes()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3. `list_scenes()` lists all scenes in the loaded DB."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nusc.list_scenes()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Render"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's plot a lidar point cloud in an image. Lidar allows us to accurately map the surroundings in 3D."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_sample = nusc.sample[10]\n",
    "nusc.render_pointcloud_in_image(my_sample['token'], pointsensor_channel='LIDAR_TOP')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the previous image the colors indicate the distance from the ego vehicle to each lidar point. We can also render the lidar intensity. In the following image the traffic sign ahead of us is highly reflective (yellow) and the dark vehicle on the right has low reflectivity (purple)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nusc.render_pointcloud_in_image(my_sample['token'], pointsensor_channel='LIDAR_TOP', render_intensity=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Second, let's plot the radar point cloud for the same image. Radar is less dense than lidar, but has a much larger range."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nusc.render_pointcloud_in_image(my_sample['token'], pointsensor_channel='RADAR_FRONT')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also plot all annotations across all sample data for that sample. Note how for radar we also plot the velocity vectors of moving objects. Some velocity vectors are outliers, which can be filtered using the settings in RadarPointCloud.from_file()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_sample = nusc.sample[20]\n",
    "\n",
    "# The rendering command below is commented out because it may crash in notebooks\n",
    "# nusc.render_sample(my_sample['token'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or if we only want to render a particular sensor, we can specify that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nusc.render_sample_data(my_sample['data']['CAM_FRONT'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Additionally we can aggregate the point clouds from multiple sweeps to get a denser point cloud."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nusc.render_sample_data(my_sample['data']['LIDAR_TOP'], nsweeps=5, underlay_map=True)\n",
    "nusc.render_sample_data(my_sample['data']['RADAR_FRONT'], nsweeps=5, underlay_map=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the radar plot above we only see very confident radar returns from two vehicles. This is due to the filter settings defined in the file `nuscenes/utils/data_classes.py`. If instead we want to disable all filters and render all returns, we can use the `disable_filters()` function. This returns a denser point cloud, but with many returns from background objects. To return to the default settings, simply call `default_filters()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nuscenes.utils.data_classes import RadarPointCloud\n",
    "RadarPointCloud.disable_filters()\n",
    "nusc.render_sample_data(my_sample['data']['RADAR_FRONT'], nsweeps=5, underlay_map=True)\n",
    "RadarPointCloud.default_filters()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can even render a specific annotation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nusc.render_annotation(my_sample['anns'][22])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can render a full scene as a video. There are two options here:\n",
    "1. nusc.render_scene_channel() renders the video for a particular channel. (HIT ESC to exit)\n",
    "2. nusc.render_scene() renders the video for all camera channels.\n",
    "\n",
    "NOTE: These methods use OpenCV for rendering, which doesn't always play nice with IPython Notebooks. If you experience any issues please run these lines from the command line. \n",
    "\n",
    "Let's grab scene 0061, it is nice and dense."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_scene_token = nusc.field2token('scene', 'name', 'scene-0061')[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The rendering command below is commented out because it may crash in notebooks\n",
    "# nusc.render_scene_channel(my_scene_token, 'CAM_FRONT')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is also a method nusc.render_scene() which renders the video for all camera channels. \n",
    "This requires a high-res monitor, and is also best run outside this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The rendering command below is commented out because it may crash in notebooks\n",
    "# nusc.render_scene(my_scene_token)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let us visualize all scenes on the map for a particular location."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nusc.render_egoposes_on_map(log_location='singapore-onenorth')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
